Cluster databases provide location transparency to data by allowing multiple systems to serve the same database. One specific type of cluster database is the Oracle Real Application Clusters™ product, licensed by Oracle Corporation, Redwood Shores, Calif. Sets of two or more computers are grouped into real application clusters. The clusters harness the processing power of multiple interconnected computers to provide a single robust computing environment. Within each cluster, all nodes concurrently execute transactions against the same database to synergistically extend the processing power beyond the limits of an individual component. Upon the mounting of the shared database, the real application cluster processes a stream of concurrent transactions using multiple processors on different nodes. For scale-up, each processor processes many transactions. For speed up, one transaction can be executed spanning multiple nodes.
Cluster databases provide several advantages over databases that use only single nodes. For example, cluster databases take advantage of information sharing by many nodes to enhance performance and database availability. In addition, applications can be sped up by executing across multiple nodes and can be scaled-up by adding more transactions to additional nodes. Multiple nodes also make cluster databases highly available through a redundancy of nodes executing separate database instances. Thus, if a node or database instance fails, the database instance is automatically recovered by the other instances which combine to serve the cluster database.
Cluster databases can be made more highly available through integration with high availability frameworks for each cluster. The inclusion of these components provides guaranteed service levels and ensures resilient database performance and dependable application recovery. Organizationally, individual database servers are formed into interconnected clusters of independent nodes. Each node communicates with other nodes using the interconnection. Upon an unplanned failure of an active database server node, using clusterware, an application will fail over to another node and resume operations, without transaction loss, within a guaranteed time period. Likewise, upon a planned shutdown, an application will be gracefully switched over to another node in an orderly fashion.
The guarantee of service level thresholds is particularly crucial for commercial transaction-based database applications, such as used in the transportation, finance, and electronic commerce industries. System downtime translates to lost revenue and loss of market share. Any time spent recovering from a system failure is measurable in terms of lost transactions. Consequently, high availability systems budget a set time period to help minimize lost revenue due to unplanned outages. High availability systems also budget for planned service interruptions.
Three types of service outages can occur for the database service. In the first type of software outage, such as due to a termination of a database instance or a clean shutdown of hardware, a standard error message is generated when the client attempts to next use the system. If the outage closes the TCP/IP network connections (sockets) through which the client sessions are communicating, the client application will generally receive a standard error message. The client can then attempt to restart the transaction once the service is restored. This type of outage is routine and readily resolved.
In the second type of outage, the outage panics the node, which fails to close the sockets through which the client sessions are communicating. The sessions will hang when synchronous blocking read or write system calls are pending or are issued prior to the outage until the client application times out due to the TCP keep-alive timer. TCP keep-alive timers are particularly problematic for applications that require high availability and can result in extremely long delays and service interruptions. The wait for a TCP keep-alive timer can be as long as two hours. And although TCP keep-alive timers are adjustable, tuning this parameter to low values can result in false reports of failure.
In the third type of outage, the outage panics the node and fails to close the sockets through which the client sessions are communicating while no read or write system calls are pending. The sessions block depending upon whether the Internet protocol (IP) address in use is static or mobile. For sessions using static IP addresses, the sessions will wait until for a TCP timeout, which includes a tcp_ip_interval delay for connection requests and a tcp_ip_interval delay for conversation requests. The wait can be unacceptably lengthy. For sessions using cooperative resource groups with address resolution, each session will block and wait for a timeout upon issuing a read or write system call against the IP address. The session will immediately receive an end-of-file error, unblock and select the next mobile IP address in the cooperative resource group address list.
Table 1 describes the effects of service outages on a TCP/IP-based client. In the first case, an outage with sockets closed due to software failure or node shutdown, the client receives an error and recovers. In the second case, an outage with sockets left open, the client blocks and waits from 75 seconds to two hours.
TABLE 1Client Effects.State ofSockets AfterConversation (SQLBlocked in I/O ReadOutageConnection Requestor PL/SQL Request)or WriteSocket ClosedClient receives errorClient receives errorClient receives error(softwarefailure or nodeshutdown)Socket leftTcp_ip_abort_cintervalTcp_ip_abort_intervalTcp_keepalive_intervalopen (node(75 seconds)(10 minutes)(2 hours)panics)
Therefore, there is a clear need for a mechanism that can reduce the latency that is typically incurred when a socket has been left open between a first and a second process after one of the two processes has failed.